THE OPERATOR FORMULA FOR MONOTONE TRIANGLES 
SIMPLIFIED PROOF AND THREE GENERALIZATIONS 
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Abstract. We provide a simplified proof of our operator formula for the number of monotone 
triangles with prescribed bottom row, which enables us to deduce three generalizations of 
the formula. One of the generalizations concerns a certain weighted enumeration of monotone 
triangles which specializes to the weighted enumeration of alternating sign matrices with respect 
to the number of —Is in the matrix when prescribing (1,2,..., n) as the bottom row of the 
monotone triangle. 



I. Introduction 

A monotone triangle is a triangular array of integers of the following form 

On,n 

"n— l,ri— 1 On— l,n 

^3,3 • • • • • • ^3,n 

0-2,2 «2,3 • • • • • • &2,7i 

0-1,2 «1,3 • • • • • • &X,n 

which is monotone increasing in northeast and in southeast direction and strictly increasing 
along rows, that is a^j < a i+ ij + i for 1 < i < j < n, a it j < a^-ij for 1 < i < j < n 
and dij < Qjj+i for l<i<j<n — 1. Monotone triangles with bottom row (1,2, . . . , n) 
correspond to n x n alternating sign matrices, the enumeration of which (there are exactly 

71 — 1 . 

n \n+j)\ °f them) provided an open problem for quite some time, see pQ. 

j=0 

In j3] we have shown that the number of monotone triangles with bottom row (fci) . . . , kn) is 
given by 

n ad-E ks +E ks E kt ) n (i.i) 

where E x denotes the shift operator, defined as E x p(x) = p(x +1). In such an "operator 
formula" , the product of operators is understood as the composition. Moreover note that the 
shift operators with respect to different variables commute and, consequently, it does not matter 
in which order the operators in the product are applied. This formula was the basis for our 
new proof of the refined alternating sign matrix theorem [1]. 
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The purpose of this article is to present a (very much) simplified proof of (II. ip and three 
generalizations that arise quite naturally out of this new proof. Next we describe these gener- 
alizations. 

Regarding the first generalization consider the following inverse question. Given a polynomial 
r(X, Y) in X, X" 1 , Y, Y~ x (e.g., over C), find a combinatorial interpretation for the numbers 

n n ^ w 

l<s<t<n l<i<j<n 

For the polynomial r(X, Y) = id —X + XY, a combinatorial interpretation is obviously given by 
monotone triangles with prescribed bottom row; for the polynomial r(X, Y) — Y, a combina- 
torial interpretation is given by Gelfand-Tsetlin patterns with bottom row (fcj, . . . , k n ) (which 
are almost defined as monotone triangles only they need not necessarily strictly increase along 
rows) as the number of these objects is given by 

j] kj ' 1 1 . (i.3) 

l<i<j<n ^ 1 

see, for instance, [2J. Note that Gelfand-Tsetlin patterns with bottom row (k\, . . . , k n ) are in 
bijection with semistandard tableaux of shape (k n , k n -i, . . . , ki). Here, we present a combina- 
torial interpretation of (11. 2\\ for all polynomials r(X, Y) of the form 

r(X, Y) = Y + (X- id) (Y - id) s(X, Y), (1.4) 

where s(X,Y) is an arbitrary polynomial in Y", V -1 . (Obviously, (11.11) is the special 

case s(X, Y) = 1 and (11.31) is the special case s(X, Y) = 0.) Notably, this interpretation is not 
only valid for (ki, . . . , k n ) G Z n with k\ < k^ < ■ ■ ■ < k n , but also for general (ki, . . . , k n ) G IP 1 
and thus this will also lead to an interpretation of (11.11) for all (fci, . . . , k n ) G II 1 . 

The second generalization concerns the weighted enumeration of alternating sign matrices 
with respect to the number of —Is in the matrix, which was introduced by Mills, Robbins and 
Rumsey, see pj. A —1 in the alternating sign matrix translates into an entry a^j with % ^ 1 
and ai_ij_i < cii t j < (Li~i,j in the corresponding monotone triangle and, therefore, we define 
the Q-weight of a monotone triangle as Q raised to the power of the number of such entries. 

Theorem 1. The generating function of monotone triangles with prescribed bottom row (k±, . . . , k n ), 
k\ < &2 < ■ • • < k n , and with respect to the Q -weight is given by 

J] (id-(2 -Q)E ks +E ks E kt ) n hzj*. 

\<s<t<n l<«<J<n 1 

Notably, this generating function has already been introduced by Mills, Robbins and Rumsey 
in [SI Section 5]. (However, Theorem [T] provides the first formula for this generating function.) 

Finally, the third generalization is a weighted enumeration of a certain type of Gelfand- 
Tsetlin patterns (which we denote as weak monotone triangles since they include monotone 
triangles) with prescribed bottom row, which reduces to the enumeration of monotone triangles 
if we take the limit P — > 1. (That is for P = 1 the weight is 1 for monotone triangles and 
for weak monotone triangles that are not strictly increasing along all rows.) A weak monotone 
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triangle is a Gelfand-Tsetlin pattern {a>i,j)i<i<j<n with Oij-x < 0,%-ij if ? ^ 1 and % < j. The 
respective weight is defined as 

II (^-k^ou-il). 

where [statement] = 1 if the statement is true and otherwise. 

In order to state this generating function, we need to introduce the following P-generalizations 
of the difference operator A x := E x — id and the shift operator: the P-difference operator is 
defined as pA x = P~ X A X and the P-shift operator is defined as pE x = pA x + id. If we 
set P = 1 then we obtain the ordinary operators. Note that these operators commute, i.e. 
pA x pA y = pA y P A X , pE xP E y = P E yP E x and pE xP A y = P A yP E x . 



Theorem 2. The generating function of weak monotone triangles with prescribed bottom row 
(ki, . . . , k n ), ki < k 2 < • • • < k n , and with respect to the P-weight is 

p(% } n ( P E kt + P A ksP A kt ) n -p—p-. 

\<s<t<n l<i<j<n 

The paper is organized as follows. In the following section we introduce a general recursion 
and a master theorem, which implies the three generalizations all at once. In Section [3] we deal 
with the first generalization and provide the combinatorial interpretation for (11.21) if r(X, Y) 
is of the form given in (jl.4p . In Section H] we deduce the (^enumeration Q f monotone triangles 
and the P-enumeration of weak monotone triangles from the main theorem. Moreover, we 
derive some results (old and new) for the special case Q = 2 in this section. In Section we 
finally prove the master theorem. In Section [6] we discuss further projects along these lines. 



2. The recursion and the master theorem 

We define a PQ-shift operator as < pE x = Q id + pA x and a PQ-identity as p id x = Q pE x — 
pA x . For Q = 1 we have < pE x = pE x and p id x = id x and again these operators commute 
with each other and also with the operators that we have introduced before the statement of 
TheoremEl Moreover note that pE p(x) = p id p(x) = Qp{x) i£p(x) is constant (with respect 
to x). 

Let S be a finite subset of Z 2 and / : S — » C be a function. For a given (k%, . . . , k n ) G Z n 
and a function A(l\, . . . , l n -i) on we define the summation operator 

(ki,...,k n ) 

J2 a{i x , . . . ,i n -i) 

(ll,...,l n -l) 
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associated to the pair (S, f) by induction with respect to n. If n = then the application of 

fci 

the operator gives zero, for n = 1 we set ^2 A = A. If n > 2 then we define 



(-) 



(fci,...,fc n ) 



h n 1 



(fci,...,fc n _i) 



£ A(i u ...,i n ^) = \Q- iQ P E kn Q P id Ki J2 p ' n_1 E ^•••^n-i; 



(iii-A-i) 



*n-l— K n-1 



)in— 2) 



(fcl,...,fc n -2) 

+ E E P^-l^^/'^l: ■ • • 

(Zi,...,z n _ 3 ) (i,j)eS 

where here and in the following g(x) := — Yl g(x) ii a > cq and, for i < 0, 

x=a x=fe+l 



1 — K n — 1 



P 4=(id + pA a 



vj=0 



(2.i; 



(If z < then we will apply the operator pE x only to functions f(x) with pA x f(x) = for a 
j > 0, i.e. to functions for which the sum in (12. ip is in fact finite. Note that the operator pE l x 
specializes to E l x for P = 1 also if i < 0.) This generalizes the summation operator from [3], 
where we have considered the special case P = 1, Q = 1, S = {(0,0)} and f(0, 0) = — 1. 

The P-binomial coefficient is defined as 

(1 - P x )(l - P*- 1 ) •••(!- p*-"H-i) 



a; 
m 



p (l-p™)(l-p™-i)...(l-p) 

and again we obtain the ordinary binomial coefficient (^) if we take the limit P — > 1. We 
define apg(n, m, S, f; k±, . . . , A;^) inductively with respect to n: let ap^(l, m, S, f; ki) = [ k ^] p 
and 

(ki,...,k n ) 

a PjQ (n,m,S,f;k 1 ,...,k n ) = ^ a P , Q (n - l,m, S, f;h, . . . ,l n -i). 

(h,—,ln-l) 

In order to see that apg(n, m, S, f;k\, . . . , k n ) is well-defined even if S $Z Z> (i.e. the appli- 
cation of the operator pE\. to ap^(n,m, f;k\,..., k n ) makes sense even if i < 0) observe 
that 



" X " 




X 


p-rn+l 


.m. 


p 


m — 1 


P 



and 



Yp*\ x ] =y P 



yx+m pA x 



X 


= P m ( 






a 


.) 


m + 1 


p V 


m + 1 


p 


m + 1 





(2.2) 



(2.3) 



a-l 

Note that this implies g(x) = 0. 

x—a 



OPERATOR FORMULA 



implies by induction with respect to m that a Py Q(n, m,S, f;ki, . . . , k n ) is a linear combination 



of expressions of the form 



V 




"fc 2 " 






n 


P 


»~2 


P 





over C[P, P 1 ,Q,Q 1 ], and consequently (also 



by (12.2jl ). there is an z > such that pA^ap^n, m, S, f;ki, . . . , k n ) vanishes. 

Clearly, ai ) i(ra,0, {(0,0)}, —1; k%, . . . , k n ) is equal to (11. ip . Moreover, it is easy to see that 
ai 5 i(n, 0, 0, — ; k\, . . . , k n ) is the number of Gelfand-Tsetlin patterns with bottom row (k\, . . . , k n ) 
and therefore equal to (11.31) . For the general situation we will prove the following theorem. 

Theorem 3. Let n be a positive integer, m be a non-negative integer, S C Z 2 be a finite set 
and f : S — > C be a function. Then a P ^Q(n, m, S, /; k\, . . . , k n ) is given by 

pU n-^- Wm)Q -{t) Y[ ( %E kt U ks - P A ks pA h J2 f(i,j) P E k ; P E k J 

l<s<t<n y (i,j)eS 



det 

l<i ,j<n 



ki 

j - 1 + 5 jjn m 



For m = this simplifies to 



p(t)q-go jt h p E kt Q P id ks - P A ksP A kt j2 f^j)pE k ; P E ks A n pLp - 

l<s<t<n y (i,j)eS J l<i<j<n F 

3. The combinatorial interpretation of ai,i(n, 0, 5, f;ki,..., k n ) 
If we specialize P = 1, Q — 1 and m = in Theorem [3] then we obtain the following formula. 

n \E kt -A k A kt Y, f{hm:E k A n ^7 

l<s<t<n y («,j')eS / l<*<i<« Z 

By setting s(X, y) = — f(hj) Y l X J , this is equal to (11.21) if r(X, Y) is of the form given 

in (11.41) . Thus it suffices to interpret ai i i(n, 0, S, f',ki, . . . , k n ). The interpretation immediately 
follows from the recursion underlying this specialization, which reads as 

(fcl,...,fcn) k n (fci,...,fc n _i) 

A(l u ...,l n . 1 )= Yl E Mh,...,ln-X) 

(h,—,ln-l) ln—l=k n -l (ilvi'n-2) 

(ki,...,k n -2) 

n— 1 

(h,...,i n - 3 ) (i,j)eS 

For a finite subset S C Z 2 , we define an S-triangle to be a triangular array («i,.j)i<j<y<n (the 
entries are arranged in the same manner as those of monotone triangles) with the following 
properties: among the "inner" entries (aij)i<i<j<n of the triangle, a certain set of non row 
adjacent special entries is identified. If a%j is such a special entry then aj+ij and aj + i J+ i are 
said to be the parents of this special entry and we demand that there exists an s = (r, t) 6 S 
such that (a i+ ij, a i+ ij + i) = (a^+r, a it j+t). We say that the special entry a^j is associated to s. 
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Observe that the second summand in the recursion takes into account for these special entries. 
(In this case fc n _i is the special entry.) On the other hand and regarding the first summand in 
the recursion, for all a it j with i ^ 1 that are not a parent of a special entry, we demand that 
^ a i,j < a i-i,j ^ a i-ij-i ^ a i-i,j an d > a iy j > Oj_ij if aj_ij_i > Oj-ij; in the 

latter case we say that the pair (aj_ij_i, Oj-ij) is an inversion. We fix a function / : S — > C 
and define the weight of such an S-triangle as 

^ I)'*' °^ inversions J~J °^ s P cc i a l entries associated to s 

With this definition, a^i (n, 0, S, f;k±, . . . , k n ) is the sum of the weights of all S-triangles with 
bottom row (hi, ... , k n ). 

To give an example, observe that 

6 

5 9 
4 10 8 
3 4 8* 9 
7 3* 10 7 10 

is a {(0, 1), (2, 0)}-triangle, where the special entries are marked with a star. The S'-triangle 
has two inversions, one in the bottom row, i.e. (10,7) and one in the middle row, i.e. (10,8). 
Thus, if /(0, 1) = qi and /(2, 0) = q 2 then the weight of this S'-triangle is qiq 2 . 

Obviously, 0-triangles with (weakly) increasing bottom row are simply Gelfand-Tsetlin pat- 
terns. However, the notion of {(0, 0)}-triangles does not coincide with the notion of monotone 
triangles. Still ai 5 i(n, 0, {(0, 0)}, — 1; hi, . . . , k n ) is the number of monotone triangles with bot- 
tom row (k\, . . . , k n ) if ki < k 2 < ■ ■ ■ < k n . On the one hand and as mentioned above, this easily 
follows from the recursion underlying ai t \(n, 0, {(0, 0)}, —1; k±, . . . , k n ). On the other hand, this 
can also be shown using the combinatorial interpretation that we have just introduced. For 
this purpose, fix (k±, . . . , k n ) e 7L n with k\ < k 2 < ■ ■ ■ < k n . Under this assumption, a {(0, 0)}- 
triangle with bottom row (fc 1? . . . , k n ) has no inversion. It suffices to show that the weighted 
sum E with respect to / = — 1 over all {(0, 0)}- triangles with bottom (ki, . . . , k n ) that have 
at least one special entry is the negative of the number of all Gelfand-Tsetlin patterns with 
bottom row (k\, . . . , k n ) that are not monotone triangles. If we ignore the marks of the special 
entries of a {(0, 0)}- triangle with bottom row (hi, ... , k n ) that appears in the sum E then we 
clearly obtain a Gelfand-Tsetlin pattern with bottom row (k±, . . . , k n ) that is not a monotone 
triangle. Thus, we have to show that for each fixed Gelfand-Tsetlin pattern that is not a 
monotone triangle the sum of weights of all {(0, 0)}-triangles in E, whose unmarked version 
is equal to this fixed Geland-Tsetlin pattern is —1. Indeed, suppose that m is the number of 
row adjacent pairs that are equal in the fixed pattern. The candidates for the special entries 
are those entries that are situated in a row below and in between such pairs. If we mark k 
special entries in the Gelfand-Tsetlin pattern then the weight is (— l) fe and, ignoring for a while 
the constraint that the special entries are not suppossed to be row adjacent, there are clearly 
(™) possiblities to do this. However, in this calculation the "forbidden" (i.e. row adjacent) 
markings cancel each other as ajj_i = a^- = a^j + \ implies aj+ij = a it j = aj+ij+i and every 
forbidden marking that contains ai-ij-i and aj-ij but not Ojj cancels with the corresponding 
forbidden marking that includes a it j. Thus, as k ranges between 1 and m, the multiplicity in 
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question is 




1 = (1 - 1) 



in 



- 1 



-1 



and the assertion follows. However, we want to emphasize that {(0, 0)}- triangles (unlike mono- 
tone triangles) provide a combinatorial interpretation of a^ifn, 0, {(0, 0)}, — 1; k±, . . . , k n ) (and 
therefore of (11.11) ) if fcj > k i+ i for an i G {1, 2, . . . , n — 1}. 

Although we think that it is of less interest, we conclude this section by discussing the case 
that Q ^ 1, i.e. give a combinatorial interpretation of cti^n, 0, S, f; hi, . . . , k n ). Here, we have 
different requirements for the ay with i ^ 1 that are not a parent of a special entry. Again we 
demand that Oi_i j_i < a^j < cfj-ij if aj_xj-i < a i-i,j an d this entry contributes the weight 
Q if < a^j < aj_ij, the weight 1 if Oj-ij = a^- < Oi-ij or aj_ij_i < ajj = flj-ij and 

the weight 2 — Q if a^ij-i = eijj = If aj-ij-i > di-ij then ai_ij_i > a^- > and 

this entry contributes the weight — Q if ai_ij_i > cijj > cii-ij, otherwise it contributes 1 — Q. 
In this case the total weight of a fixed S'-triangle is the product of the Q-weights of its entries 

times Y\ f (^)^ °^ s P ec ' a ' en * r ' es associated to s 



4. The Q-enumeration of monotone triangles and the P-enumeration of 



First we consider the Q-enumeration of monotone triangles. We claim that this weighted 
enumeration is obtained by specializing m = 0, P = 1, S = {(0, 0)} and / = — 1 in Theorem [3j 
Indeed, in this case the recursion simplifies to 



WEAK MONOTONE TRIANGLES 



(fci,...,fc n ) 



A(h,...,l n ^) 




(fci,...,fc n _2 





(/l,...,Z„-3 




(fcl,.., 



k„-2) 





(h,...,l n -3) 



s 
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which is furthermore equal to 

(ki,...,k„) 

A{l x ,...,l n -i) 

kn-1 (fcl,...,fc„_l) (fcl,...,fc n _l) 

Z„_l=fc n _l + 1 (ll,...,l n -2) {h,--,ln-2) 

(fel,...,fen-l) (fcl,...,/c n _2) 
+ . . . ,l n ^2,k n -i) — . . . , l n -3, kn-1, k n -i). 

(h,—,ln-2) (h,---,ln-3) 

It is straightforward to check that the specialization of the formula in Theorem [3] results in the 
formula of Theorem [TJ 

Let us report on a subtility, which may be of importance when applying the ideas from [I] to 
evaluate ai 7 Q(n, 0, {(0, 0)}, — 1; k%, . . . , k n ) at (ki, . . . , k n ) = (1, 2, . . . ,i — 1, i + 1, . . . , n + 1) in 
order to study a weighted refined enumeration of alternating sign matrices, see also Section [6j 
If we relax the condition of the strict increase of the rows of a monotone triangle to a possible 
weak increase in the bottom row then fll.ip is the number of monotone triangles with bottom 
row (ki, . . . , k n ) if k± < k% < . . . < k n . However, this does not generalize to the (^enumeration 
of monotone triangle as the recursion simplifies to 

(kl,...,k n ) (fei,...,fe„_i) 

A(li,...,l n -i) = (2-Q) A(l u ...,l n - 2 ,k n -i) 

(fcl,...,/c„_ 2 ) 

n-lj k n _i) 

(il,...,/ n -3) 

if fc n _i = £;„ and not to 

(fcl,...,fc„) (fci,...,fc„_l) 

. . . , / n _i) = -A(^l, • • • , ln-2, k n -i) 

{h,---,ln-l) {h,---,ln-2) 

(kl,...,k n -2) 

n— 1) ^n— l)- 

(Zl ,...,Z„_3) 

Before we turn to the P-enumeration of weak monotone triangles, we will demonstrate 
how several 2-enumeration of alternating-sign-matrix-structures follow from the Q-generating 
function. We start with the 2-enumeration of ordinary n x n alternating sign matrices (with 
respect to the number of —Is in the alternating sign matrix), which was first established by 
Mills, Robbins and Rumsey in [8J as a corollary of their Theorem 2 (they have shown that it 

is given by 2^ 2 )) long before the ordinary enumeration was finally settled. Also our derivation 
shows that the 2-enumeration is of a much simpler nature than the ordinary enumeration. 
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Indeed, the generating function from Theorem [T] simplifies to 

n aa+EkM n kj ~ ki 



l<s<t<n l<i<j<n 

in this case. Now, the crucial fact is that the operator f][ (id +E}. s E^ t ) is symmetric in 

l<s<t<n 

ki, k 2 , • • • , k n and, consequently, Lemma 1 from [3] implies that the generation function is equal 
to P(l, 1, . . . , 1) n k ~TT, where P{X X ,X 2 , ...,X n )= U (id +X s X t ) and, therefore, 

l<*<j<« l<s<t<n 

equal to 

2® TT ^— ^. (4.1) 

A A 7 — ? 

l<i<j<n J 

(This result has already appeared implicitly in [§].) The 2-enumeration is obtained by setting 
ki = i in this formula. If we specialize (k±, . . . , fc n _i) = (1, 2, . . . , I — 1, Z + 1, . . . , n) in the 
formula for the 2-enumeration of monotone triangles with bottom row (fci, k 2 , . . . , k n _i) then 
we obtain the 2-enumeration of n x n alternating sign matrices where the unique 1 in the first 
row is in the l-th column. That is 



n l—l 

n o-«> n jh-^ n o-on^n^ n ^ 



i<«j<» l<i<j<n-l l<i<i<" 3=2+1 i=l l<i<j'<n-l J 

n_1 1 / 1 

/•n-n-i-r. . 1 / n — 1 



i=l 



(n -/)!(/-!)! V-l 



Similarly, one can reprove the 2-enumeration of (2n — 1) x (2n — 1) vertically symmetric 
alternating sign matrices by specializing (ki, k 2 , ■ ■ ■ , k n ) = (1, 3, . . . , 2n — 1) , which results in 

2 (n-l)(n-2)) ; see jg_ 

In fact, it is possible to generalize the 2-enumeration ofnxn alternating sign matrices 
to certain partial alternating sign matrices. Let a partial m x n alternating sign matrix be 
an m x n matrix with entries 0, 1, —1, where the entries 1 and —1 alternate in each row and 
column, each row sum is 1 and the first non-zero entry of each column is 1 if there is any. (Such 
objects only exist if m < n.) The well-known bijection between alternating sign matrices and 
monotone triangles shows that partial m x n alternating sign matrices are in bijection with 
monotone triangles with m rows such that 1 < k\ < k 2 < ■ ■ ■ < k m < n. Thus and by ( 14. ip . 
the 2-enumeration of these objects is given by 

2 («) n ^^=2« Yi n Xj ~ j ''_ j ~'- (4.2) 

l<ki <fea<- ■■ <k m <n l<i<j<m ^ 0<xi<X2<---<x m <n—ml<i<j<m 

As n Xj X ^ 1 is the number of semistandard tableaux of shape (x m ,x m -i, . . . ,Xi), the 

l<i<j<m 

sum on the right hand side of ( 14.21) is the number of columnstrict plane partitions with at 
most n — m columns and parts in {1, 2, . . . , m}. By the Bender-Knuth (ex-) Conjecture, see 

m . 

for instance [2], this number is equal to Jl where (a) n = a(a + 1) • • • (a + n — 1), and 
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thus the 2-enumeration of partial m x n alternating sign matrices is given by 

[n — m + i)i 



2«n 



For the P-enumeration of weak monotone triangles, the weighted enumeration is obtained by 
specializing m — 0, Q — 1, S — {(0,0)} and / = — 1 in Theorem [31 as the recursion simplifies 
to 

(fel,...,fe„) fe n — 1 (kl,~-,kn-l) 

Y A{h, . . . , i n _o = P E kn Yl plri " E • • • > k-0 

(Ji,...,Z n _i) i n _i=fe n _i (/i,...,i n _ 2 ) 

(fci,...,fcn-2) 

— ^4(^1) • • • j ln-3, k n -l, k n -l) 

(/l,...,/ n _3) 

fen-1 (fci,...,fc n _i) (fci,...,fe n _i) 

= E E + E ^l>---^n-2,fc n ) 

in— l=fen-l (ilvi'n-2) (ilj— )in-2) 

(fcl,...,fe n _2) 

n— 1 ) k n —i) 

(h,—,ln-3) 

in this case. This easily implies that atp t i(n, 0, {(0, 0}, — 1; k\, . . . , fc n ) is the weighted enumera- 
tion of weak monotone triangles with bottom row (k\, . . . , k n ) and with respect to the P-weight, 
which was defined in the introduction. Clearly, the formula in Theorem [3] simplifies to the for- 
mula in Theorem [2] if we set m = 0, Q = 1, S = {(0, 0)} and / = — 1. 

5. Proof of Theorem [3] 
For fixed 5CZ 2 and / : S — > C, we define the operator 

V x , y = Q P E x <}id y - P A X P A y f&fi P E xP E y- 

The following lemma relates this operator to the summation operator defined in Section [2j 

Lemma 1. Suppose B(li, . . . , l n -i) is a function in li, . . . , l n -\ such that for alii G {1,2,..., n— 
2} we have V kM+1 B(l 1: . . . , l n _x) = if U = l i+1 . Then 

(fei,...,fe„) 

Y P A h ...pA ln _ 1 B(l ll ...,l n-l) 

(h,...,l n -i) 

n r— 1 n 

= ^(-ir i jj «id fcs n %E lH B{k lt ...,k r . lt k r+1 ,... i k n ). 

r=l s=l t=r+l 
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Proof. We use induction with respect to n. For n = and n — 1 there is nothing to prove. 
Suppose that n > 2. Then, by definition and by the induction hypothesis, we have 



k n 1 



£ pA h ■ ■ ■ pA^Mi, ■ ■ ■ , U) = Q" 1 X„ ? id fc;u £ P ^ ^ z 



(ill — j'n-l) 



in— 1— 



n-1 



r-1 



n-1 



^(-i) r - i n n ?^ fct 5(fc 1 ,...,A V _ 1 ,fc r+ i,...A-i,in-i) 

r=l s=l t=r+l 

n-2 r-1 n-2 

+B- i r i n n x e f&fipE^pE^ 

t=r+l (i,j)eS 



r=l s=l 
P 



^fcn-l ^^-i^C" 1 ' ' ' ' ' ^r+lj • • • , &n-2j &n-lj &n-l) ) 



This is obviously equal to 



n-l 



r-1 



^(-l) r X JJ pid fes J | $E kt B(k u . . . • • • ,fc„_i,fc„) 

r=l s=l t=r+l 

n— 1 /r— 1 ra— 1 



r=l 



v s=l t=r+l 

n-2 r-1 



n-2 



+ £(-^11?^ II X((^-,^-i-X„_ 1 ?id fc; _ 1 ) 

t=r+l 

B(k±, . . . , fc r _i, A; r+ i, . . . , & n _2, rc n _i, A; n _i)) L » 

1 ^n-l - "-n-l 



r=l s=l 



If we move the (n — l)-st summand of the second sum to the first sum and combine the third 
sum with the remainder of the second sum then we see that this is furthermore equal to 



r-1 



E(- 1 ) r_1 II ? id fc 3 II iE kt B(k 1 ,...,K-i,k r+ i,...,k n - 1 ,k n ) 

r=l s=l t=r+l 

n— 2 r— 1 n— 2 

+S(- 1 ) r n ? id *. n p^kti^n-iK-i • • • ' k r -i, K+i, ■ ■ ■ , k n -2, k n -i, A;*_i)) 



r=l s=l 



t=r+l 



and the assertion follows. 



□ 



In the following lemma we use the previous lemma to apply the summation operator to a 
special B(li, . . . , l n -i). 
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Lemma 2. Let m 2 ,m3, . . . ,m n > be integers and set m\ = — 1. Then 



(kl,...,k n ) / 

e n vi 

(h,...,2n-i) \l<s<t<n-l 



) det 




/ l<*J<n-l 





r +i n ri n ^ Ldet 

A A \ -•- j. i l<j j< n 

j=2 \l<s<t<n 



rrij + 1 



Proof. Let 



B(ii,...,ln-i)= H *U ), dft , 



l<s<t<n-l 



m j+1 + ljp 



First we show that B(li, . . . , / n _i) has the property, which is presumed in Lemma [TJ For that 
purpose it suffices to show that 

(id+S hM+1 )V hM+1 B(h,---,l n -i) = 

for all i G {1,2,... , n — 2}, where S x>y is the swapping operator, defined as S Xt yf(x, y) = f(y, x). 
This assertion follows, since, on the one hand, 



/ 



\ 



n v> 



t-ls 



i l<s<i<n — 1 



/ 



is symmetric in U and U + \ and therefore commutes with Si u i i+1 and, on the other hand, 
is antisymmetric in ^ and Furthermore, by (12. 2p . we conclude that 



det 



l<i,j<n-l L m J+ 1 + i Jp 



k 

mj+i+l 



P A h ... P A ln _ 1 B(l 1 ,...,l n _ 1 )= P A h ... P A ln _i H V lt>ls det 

\ J -- L I l<i.i<n— 1 



v Ks<t<n-l 



m 3 - + i + 1 



J] V k , ls \ P A h . . . P A ln _ det 

/ Kj,i<n-1 

v l<s<i<n-l 



_m i+ i + lj p 



II ^ k-^ 



,Ks<«n-l 



h 

m j+ i + 1 



| det 




/ l<«,i<n-l 





Therefore, by Lemma [U the left-hand side of the identity stated in the lemma is equal to 

n n r— 1 n 

n^Ec-ir 1 !! ? id *. n Q P E k B{k 11 ...,K-i,kr + u...,k n ). 
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n n 



( 



n pmj E(- i ) r_i 

j=2 r=l 



\ 



n 



Ms 



l<s<t<n 



r-l 



I 



n ?' d ., n ? e 

s =l t=r+l 



det 

l<i,j<n— 1 



JTlj+i + 1 



Since 



P A fcr det 

l<i,j<n— 1 



JTlj+i + 1 



(Zl,.. .,/„_! ) = (fcl,...,fc r ,...,fc n ) 



= 0, 



(/l,...,/ n _l) — (fci ,...,fc r ,...,/c n ) 



this is furthermore equal to 



j=2 r=l 



r-l 



. l<s<t<n I s =l t=r+l 



det 

l<i ,j<n— 1 



_mj+i + 1 

= Q- n+i l[p^j2(- i r 1 ( n ^ 



(;i,---,/n-l) = (fclv,fcr,---,fcn) 



j=2 r=l \l<s<t<n 



j=2 \l<s<t<n / r=l 



| det 


h 




J l<i,j<n-l 


_m j+1 + 1 


p 


- 1 det 


" k ' 
_m j+1 + 1 




l<i,j<n— 1 


p 



(ii,...,i n _i)=(fei,...,fcT.,...,A;„) 



This is now the right-hand side of the identity in the statement of the lemma, which is evident 
when expanding the determinant in the statement of the lemma with respect to the first column. 

□ 

We define a quantity that is even more general than ap^n, m, S, f; ki, . . . , k n ). For r > 1 
and (mi, . . . , m r ) G Z> , let 



apo(l, (mi, ... , m r ), 5, f;ki,..., k r ) 
and, for n > 1, let 



]^[ V k t ,k 3 
K l<s<t<r 



) det 


ki 




, m 3. 



ap,Q{n, (mi, . . .,m r ),S, f;h,..., £:„ +r _i) 

(fci,...,fc n+r _i) 

= ^2 o>p !Q (n-l,(mi,...,m r ),S,f;li,...,l n+r - 2 )- 

(h,...,l n +, — 2) 
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By induction with respect to n, Lemma [5] shows that «p ) g(n, (m 1; . . . , m r ), S, /; k%, . . . , k n+r -i) 
is equal to 



pl(n+3r-3)(n-l)(n-2)+(m 1 +m 2 +...+m r )(n-l)Q-( n 2 ) ( j | y^^ j 

\l<s<t<n+r-l / 



det 

l<i ,j<n+r— 1 



_\j < n \{j ~ !) + [i > n]K'-fl+i +n - 1) 



J p 



The first statement in Theorem [3] is the special case r = 1 and mi = m. The second statement 
follows as 



kj 

by the g-Vandermonde determinant evaluation. 



det 

l<i j'<n 



l<i<j'<n 



6. Remarks and further projects 

The starting point for this paper was [3], where we have studied the recursion underlying the 
counting function for monotone triangles with prescribed bottom row. In the present paper, 
we have considered a generalized recursion, which we have obtained by carefully introducing a 
number of new parameter, namely m, P, Q, a finite subset 5CZ 2 and a function / : S — > C, in 
the original recursion. The analysis of this generalized recursion was possible as we have finally 
noticed that the analysis of the original recursion can be simplified significantly. 

With the exception of m, all new parameters have been used to either obtain weighted 
enumerations of monotone triangles, respectively weak monotone triangles or a combinatorial 
interpretation of a generalization of ( 11. ip . However, the parameter m is also of special interest 
since it offers the possibility to "control" top and bottom row of a monotone triangle - so 
far we were only able to "control" either row. Indeed, for fixed n > 1 and 1 < i < n, let 
( c p,g)p,(?>o be complex coefficients, where almost all coefficients are zero, such that the polyno- 
mial ^2 c p,q{ ~ P ) (i n vanishes for all k G {1,2, . . . ,n} \ {i} and is equal to 1 for k — i. 

p,g>o 9 
Then, it is not hard to see that 

^2 c P , q ai,i(ra, q, {(0, 0)}, -1; h - p, . . . , k n - p) 

P,9>0 

is the number of monotone triangles with bottom row (k%, . . . , k n ) and top row i. (Here, we 
need the fact that 

(fci,...,fc n ) (fci— p,...,k n — p) 

^2 A (h-p,h-p,---,ln-l-p) = A{l 1 ,l 2 ,---,ln-l)-) 

(h,...,l n -i) 

In a forthcoming paper, we want to use the methods from [1] to attack a certain doubly refined 
enumeration of alternating sign matrices, namely study the number ofnxn alternating sign 
matrices where the unique 1 in the top row is in column % and the unique 1 in the bottom row 
is in column j. (Cleary, this number is equal to 

^2 c p,q a iA n ~ !j9> - 1 ; 1 -P, 2 ~P, ■ ■ ■ J ~ 1 ~PJ + 1 ~PJ + 2 ~P, ■ ■ ■ > n -P)-) 

p,q>o 
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Note that this doubly refined enumeration of alternating sign matrices has already been consid- 
ered in [10J. Promising computerexperiments show that Oft^n — 1, q, {(0, 0)}, — 1; 1 — p, . . . , j — 
1 — Pi 3 : + 1 ~ Pi ■ ■ ■ > n ~ p) is in fact "round" (i.e. has only relatively small prime factors) for 
certain choices of p and q. For instance, we have worked out the following conjecture for q = 1, 



is the number ofnxn alternating sign matrices that have a 1 which is situated in the first row 
and i-th column. 

Of course, it is also of interest to apply the ideas from [3] to obtain informations on the special 
evaluations at (Aft, . . . , k n ) = (1, 2, . . . , i — 1, i + . . . + of the other generalizations of (II. ip . 
For instance, the evaluation of the Q-enumeration in Theorem[T]at (Aft, ... , k n ) = (1,2, ... ,n) is 
the weighted enumeration ofnxn alternating sign matrices with respect to the number of —Is 
in the alternating sign matrix and, similary, the evaluation at (Aft, . . . , k n ) = (1, 3, . . . , 2n— 1) is 
the weighted enumeration of [2n — 1) x [2n — 1) vertically symmetric alternating sign matrices. 
So far, there do not exist formulas for these generating functions (it is likely that it is rather 
difficult to come up with a simple formula as the generating functions do not seem to be 
"round"), however Kuperberg P, Theorem 4] proved a number of factorizations regarding 
generating functions of this type. Is it possible to use the methods from [I] to deduce refined 
versions of these relations? 

Another natural question is whether the new insight in the recursion underlying f 1 1.11) leads 
to further generalizations of the formula. A project along these lines will be the following: 
for 1 < i < n, let a monotone (i, n) -trapezoid be a monotone triangle with the first i — \ 
rows removed. We want to study the number of monotone {%, n)-trapezoids with prescribed 
bottom row (Aft, ... , k n ). Obviously, the underlying recursion is the same as those for monotone 
triangles with prescribed bottom row. The difference (and the difficulty) lies in the inital 
condition. Secondly, we want to remark that our P-enumeration of weak monotone triangles 
with prescribed bottom row is a result of our efforts to obtain a weighted enumeration of 
monotone triangles, where the weight of a given monotone triangle is equal (or at least related) 
to P raised to the power of the sum of entries. Is it possible to write down a closed (operator) 
formula for a generating function of this type, or is the P-enumeration of weak monotone 
triangles already the best we can achieve in this respect? Thirdly, it should be mentionend 
that the result regarding the combinatorial interpretation of ( 11.21) emerged during our efforts 
to find other objects whose counting functions can be expressed by an operator formula. The 
(somehow inverse) strategy, which finally led to the result, was to start with an operator 
formula and to search for objects whose weighted enumeration is given by the formula. The 
solution given in Section [3] is in the sense not satisfactory as it does not lead to a "plain" 



01,1(71 - 1, 1, {(0,0)}, -I; 1 -p,2 -p, . . . , j - 1 -p,j + 1 -p,j + 2 -p, . . . ,n - p) 




where A n = FJ ^ttv is the number of n x n alternating sign matrices and 





n-2 
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enumeration when specializing the weights and, moreover, the definition of S'-triangles is a bit 
involved. Clearly, it would be of interest to search for other operators formulas and (simpler) 
combinatorial objects that are enumerated by these formulas by varying the operators and/or 
the (factorizing) polynomial to which the operators are applied. 
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